Coordinated Plan Management Using Multiagent MDPs

For the past several years, we have been developing multiagent technology to help humans coordinate their activities in complex, dynamic environments. In recent work on the DARPA COORDINATORs program, we have developed multi-agent Markov-decision process (MDP) techniques for distributed plan management. The COORDINATORs problems arrive in distributed form, with different agents getting local views of their portion of the problem and its relationship to others. Even so, the individual agents’ MDPs that capture their local planning and scheduling problem can be too large to enumerate and solve. Furthermore, the COORDINATOR agents must build and execute their plans in realtime, interacting with a world simulation that makes their actions have uncertain outcome. Accordingly, we have developed an embedded agent system that negotiates to try to find approximately-optimal distributed policies within tight time constraints. Our work draws together and extends ideas in multi-agent Markov decision processes, real-time computing, negotiation, metalevel control, and distributed constraint optimization. Contributions of our work include “unrolling” techniques for translating local hierarchical task networks to MDPs, “informed” heuristic search control of the unrolling process, and negotiation methods for allocating responsibilities across cooperating agents and using those allocations to influence local policy construction. In the rest of this paper, we describe our approach in more detail. We begin by summarizing the challenges in distributed plan management embodied in the COORDINATORs problem, and the TÆMS representation used to model the actions and interactions requiring coordination. We then de-